[Bugfix][V1] Use the Mamba cache group's block size for align-mode chunk splitting - #54076

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

Reviewer context: this is a distinct heterogeneous-block-geometry correctness fix, not a model-specific optimization. Current main reproduces the issue with cache_config.block_size = 816 and MambaSpec.block_size = 1648; the wrong grid can lead to a Mamba state being published under a boundary it does not physically represent. The added real-module regressions fail before the patch and pass after, while equal-geometry behavior remains unchanged. The production deployment that exposed the invariant also saw immediate prefix reuse recover from 0 to 100% on the primary workload, but the upstream justification here is correctness of Mamba state geometry. Review would be appreciated.

Independent confirmation of both the defect and the fix direction, from a second deployment.

Setup: a hybrid attention+mamba model (48 layers: full-attention every 4th, GDN/linear-attention elsewhere plus one conv layer), mamba_cache_mode="align", prefix caching on, chunked prefill with max_num_batched_tokens=2048. The resolved MambaSpec.block_size is 1568; cache_config.block_size ended up 4 — dragged down by the min-over-groups assignment in engine/core.py because the model also registers a small circular-buffer KV group. So _mamba_block_aligned_split was aligning to a 4-token grid, i.e. effectively never clipping.

Observed consequences match this PR's description exactly:

1. Interior boundary slots stay null. The runner writes one state column per step, so with the split inert the stored-boundary cadence collapses to ceil(P / max_num_batched_tokens) − 1 — measured 7/10 boundaries materialized for a 15,680-token prompt and 127/167 for 262,144 tokens, with the missing boundaries exactly the ones a later prefix-reuse lookup asks for.
2. Positional hash poisoning. With the wrong grid a slot holding state@2048 is published under the hash of state@3136 (cache_full_blocks hashes positionally). Any sibling request sharing ≥2 aligned blocks of prefix can be served a truncated recurrent state with no error — reachable in production with shared system prompts, no KV connector involved.

We applied a fix identical in spirit to this PR (derive the grid from the resolved MambaSpec block size out of kv_cache_config.kv_cache_groups, with a fail-closed assert when align-mode groups disagree) and the prefix-cache hit quantization law became exact again — floor((P−1)/1568)×1568, verified bit-for-bit at prompt lengths from 1,568 to ~900K tokens, including the previously-failing exact-multiple lengths.

One addition worth folding in (or taking from #53479): the boundary stop should be unconditional. next_block_boundary if start % block_size != 0 else 0 permits a multi-block advance whenever the start is aligned and max_num_batched_tokens > 2 × block_size; since the allocator hands out at most one real mamba block per running step, a chunk spanning k blocks leaves k−1 slots permanently null even with the grid fixed. We ship both changes together and the null-slot class is gone at every max_num_batched_tokens we tested (512–16384).

Independent confirmation from a second deployment — thank you, this is exactly the kind of evidence that helps.

Your addition is correct and now folded in (2ee5edd): the boundary stop is now unconditional — next_block_boundary fires even from an aligned start, so a full-budget chunk can no longer span k blocks and leave the k−1 interior state slots null. One deliberate exemption: steps using internal prefill checkpointing (use_internal_checkpoint with an aligned start) keep the multi-block advance, because that mode materializes the interior states itself.

Verification on this branch: new regression test_aligned_start_does_not_span_multiple_state_blocks walks a 5-block prompt with the full remaining budget each step — before the change it materializes only [6592] (your failure mode exactly), after it all of [1648, 3296, 4944, 6592, 8240]. The expected-chunk-sequence tests that encoded the old conditional cadence were updated to the per-boundary cadence. Targeted suites: 177 passed (chunk-split + prefix-cache + prefix-caching); ruff clean.

Your min-over-groups report (cache_config.block_size = 4 next to MambaSpec 1568) is also a cleaner natural repro of the heterogeneous-grid reachability than the drafter-group case in the PR description — the mechanism is identical (engine core min() over group block sizes).

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wickist.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Measured on GB10 with MTP n=5: this patch alone takes healthy-acceptance turns from 44 % (12 starts) to 15/16; numbers and method in #53142 (comment).

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wickist.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Closing out the correction I promised here on 2026-09-03, and the answer is that I am withdrawing the
number rather than restating it.

What I posted on 09-02 — "this patch alone takes healthy-acceptance turns from 44 % to 15/16" — used
a healthy/broken per-turn metric that I later found measures the harness, not the patch. Our agent loop
sent ignore_eos: true, max_tokens: 130 while the model's real answer was 30–40 tokens, so every turn
continued past its end-of-turn token into one of two near-tie continuations: a chat-template restart,
which the drafter predicts at ~100 %, or a wall of <|im_start|>, which it never predicts. "Healthy" and
"broken" turns were those two filler modes. The metric is a property of ignore_eos, and it does not
measure the align defect.

Why I am not supplying a replacement effect size. I re-ran the grid EOS-correctly and have a clean
unpatched baseline — acceptance by MTP n = 60 / 58 / 46 / 40 / 37 / 29 / 26 % for n = 2..8, three
starts agreeing, and I have confirmed that arm was genuinely unpatched from the runner's own preflight
gate (it aborts if mamba_state_block_size is present in the scheduler, and it logged OK). But I have
no EOS-correct patched arm to pair it with. So there is no corrected before/after, and "the direction
stands" from my 09-03 note is not something that data supports either. Both figures are withdrawn.

What is unaffected, and is why I still think this PR is right: the defect is a code fact, not a
benchmark result. The align-mode split used the QSA ring capacity as its unit instead of the mamba
state block, so on this configuration every prefix-cache resume continued from a GDN state up to one
mamba block stale — silently, with no error. In an agent loop every turn is a resume. That argument
stands on the code and does not depend on any number I posted.

Offer. This PR has been blocked on conflicts twice in three days with no reviewer, and I would rather
hand a reviewer a real measurement than an anecdote. We have a GB10 (sm_121, TP=1) and the EOS-correct
harness. If you name the cell you want — acceptance, TTFT, or prefix-cache hit rate, patched vs
unpatched, however many starts you consider enough — we will run it and post it with the raw data,
whichever way it comes out.

Apologies for the six-day gap on a correction I said would follow immediately.

AI assistance was used in preparing this comment; the measurements are ours and were reviewed before posting.

@wickist — mechanical rather than substantive, and it may be why this has sat: DCO has been failing
since the PR opened, so CI never gets past it.

None of the three commits carries a Signed-off-by: line:

9548d605  [Test] Reproduce mamba align split with heterogeneous KV block sizes
10e28b6c  [Bugfix][V1] Use the mamba cache group's block size for align-mode chunk splitting
244edeed  [Bugfix][V1] Stop mamba align chunks at every crossed state boundary

DCO shows action_required and pre-run-check fails, which skips Check format and pre-commit
behind them. git rebase --signoff main && git push --force should clear it.

Worth flagging because DCO surfaces as a check rather than as a review comment, and it is easy to miss
while resolving merge conflicts — which you have now done twice. Fourteen comments here and no human
review submitted yet; a red DCO is a plausible reason reviewers have not picked it up.

AI assistance was used in preparing this comment.

Controlled A/B update from our side (RTX 3090 TP2, Qwen3.8 hybrid GDN + DFlash2, mamba align mode): on v0.29.0 this fix is a no-op for our configuration — the KV-config interface normalization (attention block size raised to the mamba page size, interface.py:918/942) already guarantees cache_config.block_size == MambaSpec.block_size, so the heterogeneous geometry never arises and a semantic port of this PR produced zero measurable delta (prefix-cache token-hit counts identical across arms). However, the scheduler invariant this PR adds remains valuable as defense-in-depth for configurations where heterogeneous block geometry CAN still occur (explicit --block-size, other page layouts) — the A/B now also documents precisely where the fix is and is not exercised. No changes requested; sharing the measurement for reviewer context.

Independent reproduction on a single DGX Spark (GB10, sm_121, ARM64)

We hit a deterministic, reproducible logit divergence that looks like it sits exactly in the
align-mode Mamba state path this PR touches. Posting the measurement in case it is useful as an
extra test case — we are not claiming this PR fixes it, we have not been able to run a patched arm
yet (see the note at the end).

Reproduced on two independent stacks. We ran the same probe on two images that share the checkpoint, the machine and the recipe's
two out-of-tree patches (the PLE mmap loader and patch_mamba_block_size.py), but nothing of the
engine build, and the result is identical:

Setup (all pinned, arm A figures unless noted):

The engine selects align mode automatically:

INFO [config.py:605] Mamba cache mode is set to 'align' for Qwen4ExpForConditionalGeneration
                     by default when prefix caching is enabled

The experiment. Two requests share a long identical prefix (the same document, ~3.2k prompt
tokens) and differ only in the task suffix. We compare a SHA-256 over the full per-token top-20
logprob list — text equality is not used as evidence.

So:

1. Request B on its own is perfectly deterministic across 10 runs, including its own full
   prefix-cache hits (runs 2..10 of the clean arm).
2. In the contaminated arm, B#1 — which gets a partial hit on blocks written by A and
   prefills its own suffix — reproduces the clean-arm hash bit for bit.
3. B#2..10 — full prefix-cache hits on blocks that were first written by the differently
   sized request A — diverge, starting at token 0, and then stay stable on the new value.

In other words the divergence is not "cold vs. cached"; it is whose request wrote the shared
prefix blocks. That is what made this hard to spot: within a single client's repeated traffic the
output looks perfectly stable.

Control. Same contaminated sequence with --no-enable-prefix-caching on a fresh server:
c1dc6688b1004f2f × 10 — the divergence disappears. (The absolute hash necessarily differs there:
without prefix caching the engine does not switch to align Mamba mode at all, so a different
numeric path runs. The metric is stability, not the absolute value.)

We can produce it on demand — up to a point. The log reports the cache block size:

INFO [interface.py:915] Setting attention block size to 1600 tokens to ensure that
                        attention page size is >= mamba page size.

Across the pairs we measured, the failing one is the only one where the two requests seal a
different number of full blocks (3169 → 1, 3227 → 2; the passing pairs are 1/1 and 15/15). So we
took a document that had never been used on that server start and sized the two tails to land on
either side of a block boundary:

So crossing a block boundary appears necessary but not sufficient: it reproduced the effect on a
fresh document, but a different document with the same block-count asymmetry stayed stable, and
synthetic low-entropy filler never triggered it. Content or length matters as well — the QSA sparse
block selection is our leading suspect for the missing factor, and we have not separated it out yet.

Why we care: our workload is Hungarian document extraction, where logit-level divergence has
already cost us once: in our first round, before the top-k kernel fix, run-to-run logit drift moved
extracted dates and amounts on 5 of 50 items. In the probe above the visible answer text stayed
identical (text variants: 1), which is exactly why we hash logprobs rather than text.

Important: the equivalent of both fixes is already active in the image we measured. The GB10
recipe we run (blazux/qwen3.8-Flash-DGX @ c578815)
carries a two-line out-of-tree patch that touches exactly the two files these PRs do:

mamba_hybrid.py   (new_req_data.num_computed_tokens - 1)
                  // (self.cache_config.mamba_block_size or self.cache_config.block_size)
scheduler.py      block_size = self.block_size  # scheduler block size (LCM of groups) == mamba block size

(The patch's own header describes the bug it fixes: EngineCore overwrites cache_config.block_size
with the minimum group block size — the QSA raw-key ring's 8/16 — while the align-mode state-slot
seed and the block-aligned split want the mamba block size, 1600; the result was an all-zero
restored state on every prefix hit.)

We verified both lines are present in the image under test. So what we are reporting is not the
already-known all-zero-state failure: that one is handled, and a subtler cross-request difference
remains. Your PRs do considerably more than those two lines (#54076 is +287/−17 across four files,
#53798 +64/−23 across six), so they may well cover the remaining case too — we simply cannot tell from here.

What this is not. We could not run an arm with this PR (in its full form) applied. Our image is the pinned
GB10 recipe (model package still named qwen3_8_flash_next), and vllm/vllm-openai:v0.29.0 — the
other realistic base for this hardware — also predates the current main layout. A patched arm
needs a full main build on ARM64, which for this NVFP4 checkpoint additionally depends on
#55334 and a PLE offload path (#54129 or an out-of-tree mmap loader).

Raw probe output, tooling, the exact launch commands and the full write-up:
https://github.com/k3net/docai-evals/tree/b1f14a36c5bcc02cf2cd65705031e676fa9cb73f/experiments/2026-09-12-qwen38-flash-next-prefix-cache-cross-request-gb10
(the arms in this comment are results/round3-A-t201-*.json and results/round3-B-*)

Long-form write-up: https://docai.hu/blog/prefix-cache-megvaltoztatja-a-valaszt

Follow-up to my September 12 reproduction: we now have a GPU control/patched experiment on DGX Spark (GB10, ARM64). A narrow backport of this PR's boundary-stop condition removes the within-start logprob divergence on the measured cases.

This also corrects my earlier statement that the recipe already carried the equivalent of both fixes: it fixes the block-size selection and worker seed, but does not include the boundary stop from this PR.

Setup: the same recipe-built v0.29.0 image as our previous arm B, RadixArk/Qwen3.8-Flash-Next-NVFP4, deterministic QSA top-k active, MTP=2, align-mode block size 1600, chunked prefill budget 8192, PIECEWISE, serial greedy requests. Both arms add --enable-prompt-tokens-details. The experimental scheduler change is only:

-            next_block_boundary if start % block_size != 0 else 0,
+            0 if use_internal_checkpoint else next_block_boundary,

We have not tested the full four-file PR.

The source probe, extracted from the running image, predicts a threshold at 2 × block_size = 3200 under these MTP settings: below it, the old split can run across the 1600 boundary without stopping. Whether A reaches a checkpoint boundary explains all eight previously measured pairs, including the 4689/4870 pair that contradicted our earlier block-count heuristic.

GPU threshold matrix: A once, then B four times, with a shared document prefix and separate cache salts/filler seeds per cell. PASS means one digest over the per-generated-token top-20 logprob lists within that server start; canonicalized digests agree with the verdict.

Two corrections/clarifications to the earlier report:

• B#1 is a zero hit, not the partial hit I previously inferred. The failing cell diverges when the reported hit becomes 1600 tokens on B#2.
• This is numerical, not tied-candidate reordering. For B#1 vs B#2 in the failing control cell, the first three positions have 19/20, 19/20 and 18/20 shared candidates; every shared candidate changes, with maximum absolute logprob deltas 0.74515, 0.66381 and 1.06244. The canonical digest also changes. Argmax and visible text remain unchanged in this probe.

The original, unchanged T3-01 → T2-01 reproducer (3169/3227 tokens) also passes on the patched arm: 10/10 identical logprob digests, after failing across our earlier rounds on two stacks.

Limits: the matrix uses different filler seeds per cell and arm, so it is not a literal “only nine tokens changed” A/B; the unchanged original reproducer is the stronger follow-up. Absolute hashes are not compared across server starts. Cache-hit counts remain unchanged after the patch, so the results do not by themselves prove cache provenance or tensor equality. Chunk-shape-dependent numerics are a plausible mechanism; the first differing operation still needs a tensor trace. We have not measured the backport's prefill cost, rerun the 50-item quality suite, or adopted it in production.

Scripts, raw results and experiment report.

Updated write-up: English · Hungarian.

This provides an independent numerical-stability case for the boundary-stop part of the PR, even where correcting the block-size selection alone is insufficient.

@jschmied DCO fixed — all four commits now carry Signed-off-by (rebased + force-pushed). Apologies for letting that sit since your 09-09 note; it was missed in our comment sweep, not ignored.

Two updates since:

1. New follow-up commit: scope the prefix-cache last-block drop to eagle-family drafters (use_eagle_preserves_target_kv_cache(); dflash/dspark draft from their own KV and never write target blocks, so the EAGLE volatile trailing-block drop must not apply to them — with it, the final block-aligned mamba state never materializes and prefix-cache/offload lookups collapse, [Bug]: [SpecDecode] Hybrid Mamba (align) corrupts under speculative decoding when a KV connector is attached — even with zero retrieved tokens #53505). On current main every scheduler-side consumer (the KVCacheManager use_eagle bit, the align-split back-off, drop_eagle_block) already routes through use_eagle_block_drop(), so the one predicate scopes all of it. It composes with the boundary-stop: without the scoping, the eagle back-off reintroduces exactly the one-block shortfall the boundary stop removes.

2. Your measurement offer — yes, please. The cell we'd value most: prefix-cache hit rate (cached tokens on an immediate same-prompt re-ask), patched vs unpatched, EOS-correct harness, 3 starts. If it's cheap to stack one more arm, the same patched tree + the scoping commit is what our production deployment runs (together with the boundary stop), and matches the region @k3dani validated on GB10.

AI assistance was used in preparing this comment; the changes and measurements referenced are ours and were reviewed before posting.